27: Maximum Likelihood

Today: Maximum Likelihood

Goal: Modify distribution parameters based on observed data

Objectives:

• derive maximum likelihood estimate for the exponential distribution
• derive maximum likelihood estimate for the Poisson distribution

Notation

Notation

Recall,

• Lower-case $\{x_1,x_2,x_3,...,x_n\}$ is a set of observations
• Upper-case $\{X_1,X_2,X_3,...,X_n\}$ is a set of random variables (i.e. a data set)
• Treating $\{X_1,X_2,...,X_n\}$ as a set of $n$ i.i.d. (independent and identically distributed) random variables is a common assumption.
• With independence, $$P(X_1,X_2,...,X_n)=P(X_1)\cdot P(X_2)\cdot ...\cdot P(X_n)$$
• Each individual probability is computed (at least theoretically) with a PDF (probability density function) $$P(x_i)=f_X(x_i)$$

Likelihood

::::: {.panel-tabset}

Definition

Likelihood Function

Let the likelihood function, in terms of a parameter $\theta$, be the joint probability

$$L(\theta )=P(X_1=x_1,X_2=x_2,...,X_n=x_n)=f_X(x_1)\cdot f_X(x_2)\cdots f_X(x_n)$$

or

$$L(\theta ;{\left\{x_i\right\}}_{i=1}^n)={\displaystyle \prod _{i=1}^n{f}_X(x_i)}$$

Maximum Likelihood

Definition

Likelihood Function

Given a data set $\{x_1,x_2,...,x_n\}$, we seek the desired parameter(s) that makes realizing the data set most likely.

$$L(\theta ;{\left\{x_i\right\}}_{i=1}^n)={\displaystyle \prod _{i=1}^n{f}_X(x_i)}$$

Maximization

From calculus, recall that the main step in maximizing the value of a function is setting the first derivative equal to zero.

Visuals

Example 1

Given a data set $\{x_1,x_2,...,x_n\}$, assume an $\text{Exp}(\lambda )$ distribution.

• Compute the value of rate parameter $\lambda$ that maximizes the likelihood of the data set.
• Compute the likelihood at the maximum likelihood estimate (MLE).
• Characterize the top 5 percent of light bulbs.

Example 2

Given a data set $\{x_1,x_2,...,x_n\}$, assume an $\text{Pois}(\lambda )$ distribution. Compute the value of parameter $\lambda$ that maximizes the likelihood of the data set.

Estimators Revisited

If we sample from a theoretical $U(0,M)$ distribution, the sample maximum $s_M$ of each sample is less than or equal to $M$

$$s_M\le M$$

It would follow that the average of the sample maxima underestimates the true maximum

$$\text{E}[s_M]\le M$$

Therefore the sample maximum is a biased estimator of the true maximum.

$$$$

Similarly, the sample minimum $s_m$ from a $U(m,0)$ distribution overestimates

$$\text{E}[s_m]\ge m$$

Therefore the sample min-mum is a biased estimator of the true minumum.

Looking Ahead

Final Exam will be on May 6

tweet source